-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Export Project to HydroShare #2570
Conversation
Apparently dev is out of date and beta is a better reflection of the current state of production. Based off of this comment: hydroshare/hydroshare#2530 (comment) From cursory observation, it seems like beta uses a clone of the production database, while dev is its own environment. Thus, beta uses the same OAuth credentials as production, instead of its own set like dev.
@azavea-bot rebuild |
I can't seem to connect to the internal CI server, so will see if I can investigate from another computer. |
Ah I didn't add |
b8649d1
to
56f5067
Compare
Updated credentials for staging and CI from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works as described. Nicely done, and a nice, clear chain of commits made reviewing this substantial PR a breeze.
I left a few suggestions for improvement, but they are not blocking.
src/mmw/apps/export/views.py
Outdated
project=project, | ||
resource=resource, | ||
title=params.get('title', project.name), | ||
url='{}resource/{}'.format(HYDROSHARE_BASE_URL, resource), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the HydroShare URL changes, all of our url
s will have to be updated with a data migration.
A SerializerMethodField may be an ideal alternative. Something like:
class HydroShareResourceSerializer(serializers.ModelSerializer):
url = serializers.SerializerMethodField()
class Meta:
model = HydroShareResource
def url(self, obj):
return '{}resource/{}'.format(HYDROSHARE_BASE_URL, obj.resource),
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great idea, will switch to that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ended up going with a property on the model instead, which will allow its use elsewhere in the backend if we ever need it:
commit 339d70e59a44e87d4fde7ebdf053ff9488272df8
Author: Terence Tuhinanshu <ttuhinanshu@azavea.com>
Date: Thu Dec 28 13:12:47 2017 -0500
squash! Add HydroShareResource model, serializer, migration
diff --git a/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py b/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py
index a8a18272..6705cef5 100644
--- a/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py
+++ b/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py
@@ -17,7 +17,6 @@ class Migration(migrations.Migration):
('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
('resource', models.CharField(help_text='ID of Resource in HydroShare', max_length=63)),
('title', models.CharField(help_text='Title of Resource in HydroShare', max_length=255)),
- ('url', models.CharField(help_text='URL of Resource in HydroShare', max_length=1023)),
('autosync', models.BooleanField(default=False, help_text='Whether to automatically push changes to HydroShare')),
('exported_at', models.DateTimeField(help_text='Most recent export date')),
('created_at', models.DateTimeField(auto_now_add=True)),
diff --git a/src/mmw/apps/modeling/models.py b/src/mmw/apps/modeling/models.py
index 039ab2db..d268edf5 100644
--- a/src/mmw/apps/modeling/models.py
+++ b/src/mmw/apps/modeling/models.py
@@ -5,6 +5,9 @@ from __future__ import division
from django.contrib.gis.db import models
from django.contrib.auth.models import User
+from django.conf import settings
+
+HYDROSHARE_BASE_URL = settings.HYDROSHARE['base_url']
class Project(models.Model):
@@ -110,9 +113,6 @@ class HydroShareResource(models.Model):
title = models.CharField(
max_length=255,
help_text='Title of Resource in HydroShare')
- url = models.CharField(
- max_length=1023,
- help_text='URL of Resource in HydroShare')
autosync = models.BooleanField(
default=False,
help_text='Whether to automatically push changes to HydroShare')
@@ -124,5 +124,10 @@ class HydroShareResource(models.Model):
modified_at = models.DateTimeField(
auto_now=True)
+ def _url(self):
+ return '{}resource/{}'.format(HYDROSHARE_BASE_URL, self.resource)
+
+ url = property(_url)
+
def __unicode__(self):
return '{} <{}>'.format(self.title, self.url)
diff --git a/src/mmw/apps/modeling/serializers.py b/src/mmw/apps/modeling/serializers.py
index 78d1e725..b73c18a3 100644
--- a/src/mmw/apps/modeling/serializers.py
+++ b/src/mmw/apps/modeling/serializers.py
@@ -67,6 +67,8 @@ class HydroShareResourceSerializer(serializers.ModelSerializer):
class Meta:
model = HydroShareResource
+ url = serializers.ReadOnlyField()
+
class ScenarioSerializer(serializers.ModelSerializer):
keywords = this.ui.keywords.val().trim(); | ||
|
||
if (title === "" || abstract === "") { | ||
return; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the user does not receive any feedback letting them know that these fields are required the export button appears to do nothing if both fields are not filled in. A static "required" label near fields may be all we need.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
56f5067
to
a46cb31
Compare
When we export a project to HydroShare, we need to keep track of its Resource id, title, date last exported, URL, and whether or not to autosync. This, along with standard timestamps, is tracked in a new HydroShareResource model, which has a 1-to-1 correspondence with the Project model. A Project can have 0 or 1 HydroShareResource, but a HydroShareResource belongs to exactly 1 Project. In HydroShare, the Resource ID (corresponding to the `resource` field here) is a UUID. However, we store it as a Char(63) to allow for possible changes in the HydroShare platform, as well as to maintain representational consistency. HydroShare uses small case, no-hyphen format for them, whereas Postgres and Django use the small case, hyphenated format for UUIDs. The `title` will be populated when exporting for the first time. If it is changed in HydroShare afterwards, we will not know about it since there is no hook in HydroShare to listen to. The `url` property is dynamically generated using the configured HYDROSHARE_BASE_URL and the resource ID.
We will use fiona for converting GeoJSON to Shapefile in the backend. wellknown will be used to convert GeoJSON to WKT in the frontend.
Add a new app and top-level url family for housing exports. So far we have only one: HydroShare, available at `/export/hydroshare`, but if there are more in the future they can all be grouped under `/export/`. Currently we only support exporting saved Projects to HydroShare, thus the Project must be specified in the call as so: `/export/hydroshare?project=XX`. If this project does not exist or does not belong to the user, we return a 404. The endpoint supports GET, POST, and DELETE. GET simply returns the HydroShareResource object corresponding to the project if one exists. POST for a project that already has an export currently behaves the same as GET. This will be amended in the future to update the HydroShareResource and re-export to HydroShare. POST for a project that has not been exported creates a resource in HydroShare with the given title, abstract, keywords, and any supplied files. The area of interest of the project is saved as a GeoJSON and as a Shapefile in HydroShare. Any additional files coming from the client as {name: "string", contents: "string"} objects are also added. A HydroShareResource object is created for this project and returned. DELETE removes the resource from HydroShare and also the matching HydroShareResource object from MMW.
Currently, the CSV Download in the UI is handled via the tableExport library which converts a rendered HTML table to CSV and downloads the file. Since when we're exporting from the MultiShareModal we will not have the renderings at hand to reference, we must work off the internal data each Analyze Task has. This generic implementation gets a list of column headings from the keys of the first result, and then uses the values of every result to get the rows. They are then stitched together with commas and newlines to make a valid CSV. Care is taken to quote every value, so empty values are aligned with the right columns. `null`s are replaced with empty string, and GeoJSONs are converted to WKTs.
This will be triggered from the MultiShare Modal before exporting to HydroShare. This modal will allow the user to pick a title (auto-populated with the Project's name), write an abstract (required), and supply some optional keywords. If the user cancels out of this dialog, we do nothing. Otherwise we trigger an `export` method with the values of the various fields, to be used by the calling Multi Share Modal.
The following behaviors are added: * Turning on HydroShare for a project that doesn't support it: - Starts a spinner - Opens the HydroShare Export Modal - Waits for that modal to close - If that closes successfully, then + The values are combined with a list of Analyze CSV exports and POSTed to the /export/hydroshare?project=XX endpoint + Once that succeeds, the HydroShare intro blurb is replaced with a description containing title of and link to the resource in HydroShare. - If user cancels out of it, then the spinner is stopped and the switch turned back off * Turning off HydroShare for a project that does have it: - Starts a spinner - Opens a confirmation modal letting the user know that their action cannot be undone, and that the resource will be deleted for real. - If that closes successfully, then + A DELETE is sent to /export/hydroshare?project=XX + The title / link to the resource is replaced by the intro blurb for HydroShare - If the user cancels, the spinner is stopped and the switch turned back on
Prefer using absolute paths to relative paths
a46cb31
to
989c914
Compare
I'm merging this since all the tests are passing on my local and in the interest of having something demo-able up on staging for RRP. |
Overview
This PR sets up backend and frontend components that allow the user to export a project to HydroShare. When a logged in user opens the MultiShare view in a saved project, they see the HydroShare Export switch. When this switch is turned on we first check to see if the user has authorized HydroShare. If not, we ask them to login and go through the OAuth authorization dance. With the authorization, we open a modal asking for the user to specify a title (defaulting to the project's name) and abstract, and optionally some keywords. When submitted, these values are combined with a new CSV export of each analysis type. Nothing from the modeling stage is currently included.
The above payload is sent to a new endpoint which verifies that the project exists and is owned by the user, then creates a resource in HydroShare with the given title, abstract, keywords, and files. It also adds a GeoJSON and Shapefile for area of interest of the project.
When this completes, the frontend updates to show a link to the project. Turning the switch off at this point opens a confirmation modal informing the user of the irreversibility of their action, pending which the resource is deleted from HydroShare.
This is a first draft, and there will be a number of follow up cards from this.
Connects #2544
Connects #2557
Demo
Notes
We switch from using
dev.hydroshare
tobeta.hydroshare
on the suggestion of HydroShare developers.beta
shares credentials withproduction
, so our secrets file will have to be updated correspondingly.In case an analysis fails or is still running when the export is done, it will not be included in the export.
The actual export takes quite some time: 10-15s. I'm not sure which step is the biggest bottleneck, I suspect it is the Shapefile generation, but haven't tested for sure. We can consider breaking this off into a Celery task with some persistent UI spinner to show this status outside of the Share modal.
The CSVs generated for the Analyze tabs are markedly different from the ones retrieved through the UI. This is because the UI table CSVs work off rendered tables, however this is happening in the background and thus we work with raw model data. We can make these match in a future enhancement.
A number of outstanding issues remain, which will be covered in later cards. A full list of these can be seen in HydroShare Export Planning #2507. Briefly, they are:
/export/hydroshare?project=XX
for an existing project should update that resourceI'll be creating cards for these shortly.
Testing Instructions
Check out this branch, reprovision
app
, run migrations, and bundle:Go to https://beta.hydroshare.org/ and log in with credentials in the
HydroShare OAuth MMW Beta
LastPass site.Go to :8000/, log in, and create and save a project
Go to the share menu, and enable HydroShare Export
Authorize MMW
In the Export modal, verify that the title matches the project name. Fill in an abstract, and optionally some keywords. Click "Export"
Wait for the export to finish. When it does, there should be a link to the resource in HydroShare. Click it, and ensure it has everything advertised.
Try downloading some files and ensure they work as expected.
Go back to MMW, and switch off HydroShare. Once that completes, ensure that the resource has been deleted from HydroShare.